Repository navigation
Board: totals across requests rank only under Throughput; rig pages chart decode vs concurrent requests - #141
Merged
Merged
Conversation
…hart decode vs concurrent requests (#130) - setups: a Throughput setup for results with concurrency > 1 and decodeScope total; All results keeps per-request numbers only - supabase: totals never count as a rig's, part's, runtime's or build's best decode speed, nor in top results or the rig chart - lib/streams + StreamsChart: group a rig's results by model, quant, runtime, build and part into total and per-request curves against request count; rig page draws them - guidance, CONTRIBUTING and results/README: record concurrency and decodeScope instead of batch size; totals rank under Throughput - tests: series grouping, the Throughput setup, and a total that stays off All results and the best-decode summaries
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
No submission files to import. |
The rig page's share line and the hardware page's best-per-runtime chart still counted results with decodeScope total, so rig 21 shared 'Best 627 tok/s' (144 requests combined).
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #135/#136 (issue #130, step 3). A result measured across several concurrent requests with
decodeScope: totalis the machine's throughput, not what one request sees, yet on All results (the default tab) it ranked against single-request numbers and set a rig's "best decode tok/s". With throughput submissions now on the site (e.g. 627 tok/s across 144 streams on a 3-NUC rig next to 33 tok/s single-stream), that misrepresents inference speed.What changes
concurrency > 1anddecodeScope: totalagainst each other. Rules shown:≥ 2 requests,total across requests. Per-request multi-stream results keep ranking in 4 requests as before.rank.setup: 'throughput'). LegacybatchSizeresults are untouched (the field never had a fixed meaning)./guidelines, CONTRIBUTING,results/README.md): recordconcurrencyanddecodeScoperather than "set the batch size"; totals rank under Throughput; several request counts of the same setup draw a curve.Tests
scripts/tests/streams.test.mjs(new): seeding of both curves from a single request, totals vs per-request separation, best-of repeats at a count, legacy/hidden/ambiguous exclusions, series keys, and the Throughput matcher.scripts/tests/board-submissions.test.mjs: an 8-request total (500 tok/s) stays off All results and Standard, ranks alone on Throughput, is counted insetupCounts, and never becomes the rig's best decode.npm run typecheck,npm run build,results:test(90),agents:test(25) all pass.Data
Rig 8's Inkling concurrency survey (15 points, 1–176 streams) recorded its stream count in the legacy
batchSizeonly; I am backfillingconcurrencyanddecodeScope: totalon those results through the agent API so the rig page draws the curve once this merges.